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LONG-TERM  GOALS 


As  part  of  a  phased  approach  toward  an  Integrated  Ocean  Observing  and  Prediction  System,  NOPP1  in 
1999  selected  a  few  preliminary  steps  for  development.  One  of  these  steps  was  to  plan  and  implement 
“a  community-based  'system'  to  broaden  and  improve  access  to  ocean  data”  .  The  project  described 
herein  was  proposed  in  response  to  the  2000  NOPP  BAA  and  has  as  its  long-tenn  goal  the 
development  of  such  a  system;  i.e.,  the  implementation  of  a  network  based  system  that  would  provide 
for  the  discovery  of  and  seamless  access  to  oceanographic  data.  The  ultimate  goal  is  a  system  that  will 
provide  immediate  access  to  a  vast  array  of  real-time  and  historical  oceanographic  archives  by  all 
segments  of  the  oceanographic  community  from  the  one-time  user  to  the  modelers  and  managers 
requiring  regular  and  automatic  ingestion  of  the  highest  quality  data  and  data  products. 

OBJECTIVES 

1.  The  development  of  a  data  access  protocol  capable  of  handling  the  exchange  of  all  oceanographic 
data  types  -  biological,  physical,  chemical  and  geological  -  as  well  as  products  derived  from  these 
data.  This  objective  is  based  on  the  assumption  that  at  the  core  of  any  network  based  distributed 
data  system  is  middleware  that  allows  for  the  seamless  exchange  of  data  between  system  elements. 

2.  The  development  of  a  representative  set  of  data  servers  and  data  clients  based  on  the  protocol. 
These  system  elements  will  be  used  to  evaluate  the  protocol  as  well  as  to  provide  a  catalyst  for  the 
use  of  the  system. 

3.  The  linking  of  in  excess  of  two  terabytes  of  oceanographic  data  held  in  over  250  databases  at 
approximately  40  sites  via  this  system  -  to  provide  a  core  set  of  data  in  the  system  to  stimulate  its 
use. 

4.  The  establishment  of  a  diverse  group  from  the  ocean  data  user  community  committed  to  the 
success  of  the  system. 

5.  The  building  of  the  basic  infrastructure  required  to  couple  data  providers  and  users  in  the  science 
community  with  data  providers  and  users  in  the  GIS  community. 

APPROACH  AND  WORK  PLAN 

As  requested  in  the  BAA,  the  approach  to  this  project  was  to  be  taken  in  two  phases: 

1 .  In  the  first  year  community  consensus  was  to  be  sought  for  the  basic  system  and  the  system 
was  to  be  designed  at  a  high  level,  and 

2.  In  the  second  and  third  years,  the  system  conceived  in  the  first  year  was  to  be  implemented. 

In  the  first  year  there  were  to  be  five  regional  workshops  run  by  each  of  the  five  regional  coordinators. 
The  recommendations  of  these  regional  workshops  were  to  provide  the  starting  point  for  a  system 
design  and  implementation  plan  to  be  undertaken  at  a  national  meeting  of  the  project  Co-Is  and 
secondary  investigators.  In  the  remaining  years,  a  set  of  pilot  projects  designed  to  demonstrate  use  of 
the  system  as  well  as  to  test  the  system  were  to  be  undertaken.  These  pilots  were  to  be  undertaken  on  a 
regional  basis  and  directed  by  regional  coordinators.  The  regional  coordinators  were  also  to  address 
population  of  the  system  in  their  region.  J.  Gallagher  was  to  act  as  the  project’s  technical  lead, 
coordinating  all  of  the  technical  efforts  and  R.  Chinman  was  to  act  as  project  coordinator  under  the 
direction  of  the  PI  P.  Cornillon.  Chinman  resigned  in  the  second  year  and  was  replaced  with  P. 
Hemenway  of  the  University  of  Rhode  Island. 


Acronyms  used  in  this  report  are  defined  at:  httn://nvods.org  at  the  bottom  of  the  page. 

2 

“  Fiscal  Year  2000  National  Ocean  Partnership  Program  BAA  (As  published  in  Commerce  Business  Daily,  September  16, 
1999.) 


Over  the  course  of  the  project,  M.  Davidson  was  replaced  by  A.  Ball  as  southeast  regional  coordinator, 
A.  Andren  dropped  out  of  the  project  as  the  Great  Lakes  coordinator  (and  was  not  replaced)  and  S. 
Weisberg  was  added  as  the  southern  California  regional  coordinator.  T.  Habennann  was  to  be 
responsible  for  an  interface  between  the  system  and  existing  GISs. 

WORK  COMPLETED 

1 .  In  the  first  year  planning  stage  4  regional  workshops,  one  topical  workshop  and  a  National  Meeting 
involving  all  of  the  Partners  were  completed.  The  recommendations  from  these  meetings  are 
available  at:  http://nvods.org/  (under  "Reports  and  Presentations").  These  recommendations  formed 
the  basis  for  the  work  undertaken  in  the  remainder  of  the  project.  The  resulting  system  is  referred 
to  as  NVODS,  the  National  Virtual  Ocean  Data  System. 

2.  Development  of  the  data  system  core  infrastructure: 

a.  Three  versions  (3 .2-3 .4)  of  the  NVODS/OPeNDAP3  C++  core  software  have  been  released  with 
each  incorporating  major  additions  and  corrections  relative  to  the  previous  version.  A  complete 
history  of  the  software  is  available  at:  http://unidata.ucar.edu/packages/dods. 

b.  In  addition,  a  JAVA  version  of  the  core  software  has  been  implemented  and  released. 

c.  A  specification,  based  on  XML  and  web  services,  has  been  written  for  the  protocol  -  DAP  4.0. 

d.  An  “Aggregation  Server”  has  been  developed.  It  allows  multi-file  array  data  sets  to  be  viewed  as 
single  data  objects,  thus  dramatically  simplifying  requests  for  and  return  of  the  data. 

e.  Extensive  modifications  have  been  made  to  the  Live  Access  Server  (LAS)  at  PMEL. 

f.  An  Ancillary  Information  Services  (AIS)  component  has  been  added  to  the  core  software.  This 
allows  for  the  addition  of  sites  that  provide  a  consistent  view  of  the  metadata  associated  with 
data  in  the  system  and  for  these  sites  to  be  seamlessly  accessed  along  with  the  data. 

3.  Representative  data  servers  and  data  clients  developed  and/or  upgraded  as  part  of  this  project: 

a.  Clients  -  netCDF,  HDF4,  HDF5,  FreeFonn,  JGOFS,  Matlab4,  DSP  and  JDBC. 

b.  Servers  -  netCDF  client  library,  Matlab  and  IDL. 

4.  Data  provided  by  the  system:  A  fairly  complete  list  of  data  served  via  the  system  is  available  at: 
http://www.uni data.ucar.edu/cgi-bin/dods/datasets/datasets.cgi?xmlfilename=datasets.xml.  This  list 

includes  data  entered  as  part  of  the  proposed  effort  as  well  as  data  entered  by  “outside”  groups 
referenced  in  section  5.a.i  below. 

5.  Establishment  of  a  diverse  ocean  user  community  has  been  addressed  from  two  perspectives: 

a.  Active  recruitment  of  data  providers  and  software  developers  beyond  the  group  originally 
identified  in  the  proposal. 

i.  Data  providers  from  “outside”  the  project  include:  TPAC-Australia,  U.  Hawaii,  Caro-COOPS, 
JPL  and  MIT-ECCO,  COLA,  CDC,  NODC,  NCDC  and  U.  Kansas-OBIS  Hexacoral  data. 

ii.  Groups  “outside”  the  project  making  significant  software  contributions  to  the  system: 

1 .  COLA  has  developed  the  GrADS-DODS  Server  which  includes  a  powerful  server-side 
computational  capability  as  well  as  the  ability  to  serve  GRIB  and  BUFR  data. 

2.  NCAR  has  developed  a  grid-FTP  version  of  the  core  software. 

3.  A  number  of  groups  have  developed  NVODS  clients  -  ncBrowse,  VisAD,  IDV,  etc. 

4.  Several  large  projects  have  begun  making  use  of  OPeNDAP:  GODAE,  NOMADS,  ESG-II 

b.  Four  pilot  projects  have  been  undertaken  to  help  entrain  users.  These  projects  are  focused  on  the 
data  management  community. 


3  The  data  access  protocol,  at  the  core  of  the  National  Virtual  Ocean  Data  System  (NVODS),  has  been  developed  and 
maintained  by  the  Open  source  Project  for  a  Network  Data  Access  Protocol  Inc.  (OPeNDAP),  a  non-profit  corporation 
established  explicitly  for  this  purpose.  NVODS  core  software  will  be  referred  to  as  OPeNDAP  in  this  report. 


i.  SCCWRP  Pilot,  Steven  Weisberg  -  A  water  quality  management  project  in  Southern  CA. 

ii.  Shrimp  Pilot,  Linda  Mercer  and  Philip  Bogden  -  A  fisheries  management  project  in  the  Gulf 
of  Maine  shrimp  fishery. 

iii.  Gulf  of  Mexico  Pilot,  Worth  Nowlin  -  A  project  to  make  oil  spill  model  predictions  available 
to  response  team  managers  in  the  Gulf  of  Mexico. 

iv.  CODAR  Pilot,  Dave  Ullman  -  A  project  to  provide  a  consistent  view  of  CODAR  data 
collected  from  multiple,  separately  managed  sites. 

6.  Several  projects  have  been  undertaken  to  provide  GIS  access  to  OPeNDAP-enabled  data  servers. 

a.  With  the  requisite  geo-referencing  metadata  OPeNDAP  data  objects  may  be  converted  to  a 
GeoTIFF  file  using  NOAA/NGDC  software.  These  files  can  be  read  by  most  GIS  packages. 

b.  A  GDAL-OPeNDAP  driver  for  array  data  and  an  OGR-OPeNDAP  driver  for  point  and  vector 
data  have  been  implemented  for  use  in  MapServer,  a  web-based  GIS. 

c.  As  part  of  the  SCCWRP  pilot  an  interface  to  EASy,  a  GIS,  has  been  written  for  array  data. 

d.  An  ArcIMS-OPeNDAP  connection  has  been  developed  for  array  data. 

RESULTS 

1.  Infrastructure: 

a.  The  Ancillary  Information  Services  capability  to  add,  at  a  remote  site,  metadata  associated  with 
a  data  set  is  seen  as  a  major  step  in  dealing  with  the  lack  of  metadata  consistency  in  a  highly 
distributed  data  system  based  to  a  large  part  on  voluntary  contributions  of  data.  The  AIS  has 
proven  to  be  a  major  factor  in  bringing  NVODS  data  sets  into  compliance  with  standards 
required  by  standard  software  packages  such  as  MapServer. 

b.  The  Aggregation  Server  has  proven  to  be  a  significant  first  step  in  simplifying  access  to  large 
multi-file  data  sets.  Work  to  address  aggregation  of  multi-file  in  situ  data  sets  has  begun  with  a 
project  sponsored  workshop  (January  2004)  addressing  aggregation  of  mooring  data. 

2.  Population  of  the  system  is  growing  rapidly  with  more  than  500  data  sets  now  accessible  via 

OPeNDAP  servers  at  more  than  40  institutions  worldwide.  These  servers  are  responding  to  over 
2  million  individual  data  requests  per  month  from  approximately  2000  users  and  result  in  the 
transfer  of  over  200  Gbytes  of  data  per  month.  The  rapid  growth  in  the  number  of  data  providers 
and  data  users  suggests  that  the  system  has  reached  a  point  at  which  it  can  continue  to  grow 
without  additional  seed  funding;  i.e.,  the  population  effort  funded  as  part  of  this  project.  Of 
particular  interest  here  is  the  length  of  time  required  for  the  adoption  of  new  technologies.  As 
observed  in  other  similar  endeavors,  it  appears  that  on  the  order  of  ten  years  of  development  and 
implementation  is  required  for  a  significant  system  component  to  be  widely  adopted;  six  years  of 
funding  from  other  sources  -  NASA,  NOAA  and  NSF  -  received  for  DODS  prior  to  the  NOPP 
funding  reported  on  herein  was  used  to  develop  the  base  on  which  NVODS  was  built.  Figs.  1-3 
show  usage  statistics  for  two  data  providers  for  the  past  28  months.  The  number  of  data  accesses 
system-wide  (not  shown)  has  grown  by  approximately  1 15,000  per  month  over  the  past  2  years. 

IMPACT  AND  APPLICATIONS 
National  Security 

The  potential  for  use  of  the  system  to  access  environmental  data  for  National  Security  related  issues  is 
expected  to  grow  with  the  volume  and  breadth  of  data  available  via  the  system. 
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Figure  1.  Data  accesses  per  month  via  OPeNDAP-GDS  servers  at  COLA  and  UCAR. 
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Figure  2.  Number  of  unique  IP  addresses  per  month  accessing  data  via  OPeNDAP-GDS  servers  at 
COLA  and  UCAR. 
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Figure3.  Data  volume  in  gigabytes  accessed  from  OPeNDAP-GDS  servers  at  COLA  and  UCAR.  The 
anomaly  in  April  2003  resulted  from  one  user  downloading  a  major  archive  via  the  system.  The  system 
was  subsequently  modified  to  prevent  such  use  in  the  future. 

Economic  Development 

NVODS  components  are  expected  to  be  adopted  as  elements  of  future  data  systems.  Adopting  system 
components  will  result  in  major  time  and  associated  dollar  savings  in  the  development  of  new  systems. 

Science  Education  and  Communication 

Facilitating  access  to  data  is  at  the  core  of  the  NVODS  effort.  NVODS  is  expected  to  play  a  major  role 
in  the  communication  of  scientific  data  both  to  the  research  community  and  to  the  educational  sector. 

TRANSITIONS 
National  Security 

Analysis  of  OPeNDAP  access  logs  suggests  that  the  system  is  being  heavily  used  by  the  military  to 
access  both  meteorological  and  oceanographic  data  in  an  operational  mode. 

Economic  Development 

The  Data  Management  and  Communications  (DMAC)  committee  of  the  Integrated  Ocean  Observing 
System  (IOOS)  has  adopted  OPeNDAP  for  the  data  transport  component  of  IOOS.  The  core  software 
is  also  being  used  in  a  number  of  other  large-scale  projects:  NOMADS,  GODAE,  ESG  II, 
PARADIGM,  etc.  In  addition  use  of  the  system  to  help  address  resource  management  problems  in 
fisheries,  oil  spill  mitigation  and  water  quality  monitoring  is  being  explored  via  pilot  projects. 


Science  Education  and  Communication  (Delete  this  section  if  there  are  none) 

NVODS  is  heavily  used  in  the  scientific  community  to  access  oceanographic  and  meteorological  data. 

RELATED  PROJECTS 

The  nature  of  this  project  has  been  to  form  collaborations  with  a  large  number  of  other  groups.  A 
number  of  these  collaborations  have  resulted  in  a  continuing  close  working  relationship  that  are 
expected  to  continue  beyond  this  grant.  These  collaborations  fall  into  three  areas: 

1 .  Many  groups  are  providing  data  via  OPeNDAP  servers,  but  in  most  cases  these  groups  are  doing  so 
with  little  contact  with  the  project  office.  Those  working  closely  with  the  project  office  and  with 
which  we  anticipate  a  close  continuing  relationship  are:  W.  Nowlin,  Texas  A&M;  B.  Butman, 
USGS;  R.  Raskin,  JPL  PO-DAAC;  P.  Sharfstein,  FNMOC;  B.  Domenico,  Unidata;  T.  Habermann, 
NGDC;  P.  Hamilton,  SAIC;  S.  Hankin,  PMEL;  K.  Casey,  NODC,  B.  Blumenthal,  LDEO;  J.  O’Brien, 
FSU;  J.  Kinter,  COLA;  P.  Bogden,  GoMOOS;  R.  Schweitzer,  CDC;  E.  Chassignet,  U.  Miami;  F. 
Muller-Karger,  USF;  J.  Lever,  NAVOCEANO;  and,  S.  Kempler,  Goddard  DAAC. 

2.  Similarly  there  are  now  many  groups  beginning  to  develop  OPeNDAP-enabled  software,  clients, 
servers  and  specialized  functions.  Those  with  which  we  anticipate  a  close  continuing  relationship 
are:  B.  Domenico,  Unidata;  J.  Kinter,  COLA;  S.  Hankin,  PMEL;  D.  Denbo,  PMEL;  N.  Potter, 

OSU;  L.  Olsen,  GCMD;  T.  Burk,  MapServer  and,  P.  Fox,  HAO. 

3.  In  addition  there  are  several  large  projects  that  are  making  use  of  OPeNDAP  (a.k.a.  DODS)  as  a 
component  of  their  data  transport  layer:  NOMADS  http://nomads.ncdc.noaa.gov/.  GODAE 
http://www.bom.gov.au/bmrc/ocean/GODAE/Meetings/7thIGST/data  sharing  paper.doc.  DMAC/IOOS 
http://dmac.ocean.us:8101/,  ESG  II  https://www.earthsvstemgrid.org/about/overview.isp:  and,  COLA 
http://grads.iges.org/grads/gds/. 


PUBLICATIONS 

Cornillon,  P.,  J.  Gallagher  and  T.  Sgouros,  "OPeNDAP:  Accessing  Data  in  a  Distributed, 
Heterogeneous  Environment",  Data  Science  Journal,  2,  November  2003,  pp  164-174. 

CONSIDERATION  FOR  EXCELLENCE  IN  PARTNERING  AWARD  (COMPLETE  THIS 
SECTION  ONLY  IF  YOUR  PROJECT  FUNDING  FINISHED  THIS  YEAR  OR  IN  THE  LAST 
12  MONTHS,  OTHERWISE  PLEASE  DELETE  THIS  SECTION. 

1.  Ocean  Sector  Diversity:  Project  partners  come  from  all  of  the  oceanographic  subdisciplines  - 
physics,  biology,  chemistry  and  geology  -  as  well  as  from  academia,  state  government,  the 
federal  government,  industry,  and  the  international  community. 

2.  Partner  Involvement: 

a.  Mark  Abbott  -  Hosted  the  West  Coast  Regional  Workshop  in  year  1 .  Served  OSU  data  via 
the  Java  based  JDBC  server.  Moderate-to-low  participation. 

b.  Margaret  Davidson  -  Replaced  by  Anne  Ball  who  ran  the  Southeast  regional  workshop. 
Moderate-to-low  participation. 

c.  Linda  Mercer  -  Hosted  the  Northeast  Regional  Workshop.  Hosted  several  other  regional 
workshops  to  identify  a  regional  project.  Organized  and  ran  the  Shrimp  Pilot.  Served  on  the 
NVODS  Advisory  Committee.  Moderate-to-high  participation. 

d.  Worth  Nowlin  -  Hosted  the  Gulf  Coast  Regional  Workshop.  Worked  very  actively  with 
regional  data  providers  to  help/encourage  them  to  serve  their  data  via  the  system.  Very  active 


member  on  the  NVODS  Advisory  Committee.  Developed  the  Gulf  of  Mexico  Oil  Spill 
Response  Pilot.  Organized  the  Gulf  Coast  Regional  group  into  the  base  for  a  Gulf  Coast 
Regional  Observatory.  Very  high  participation. 

e.  Brad  Butman  -  Served  a  significant  volume  of  data  via  the  system.  Moderate  participation. 

f.  Richard  Chinman  -  Coordinated  the  overall  effort  in  the  first  year.  Was  replaced  by  Paul 
Hemenway  in  the  second  year.  Very  high  participation. 

g.  Don  Collins  -  Served  data  from  the  PO-DAAC  into  the  system.  Moderate  participation. 

h.  James  Cummings  -  Participated  in  the  first  National  Meeting  of  the  project.  Low 
participation. 

i.  Glenn  Flierl  -  Very  active  member  of  the  NVODS  Advisory  Committee.  Also  provided  input 
periodically  with  regard  to  system  performance  and  evolution.  Moderate -to-high 
participation. 

j.  Dave  Fulker  -  Replaced  by  Ben  Domenico  -  Provided  user  services  for  the  system.  Served 
Unidata’s  significant  near-real  time  data  holdings  via  the  system.  Developed  the  Aggregation 
Server.  Included  the  OPeNDAP  core  in  the  Java  version  of  netCDF.  High  participation. 

k.  James  Gallagher  -  Project  technical  lead.  Very  high  participation. 

l.  Ted  Habermann  -  Lead  the  effort  to  interface  system  accessible  data  to  ESRI  GIS  products. 
Also  developed  the  GeoTiff  output  engine.  Served  NGDC  data  via  the  system.  Moderate-to- 
high  participation. 

m.  Peter  Hamilton  -  Installed  servers  for  SAIC’s  extensive  mooring  archive.  Analyzed  the 
organization  of  OPeNDAP-accessible  mooring  data  and  hosted  a  workshop  to  develop  a 
common  data  model  for  these  data.  Moderate-to-high  participation. 

n.  Steve  Hankin  -  Very  active  member  of  the  NVODS  Advisory  Committee.  Actively  promoted 
the  system  -  many  of  the  non-funded  data  providers  were  entrained  into  the  system  by  him. 
Extended  LAS.  Provided  access  to  a  significant  fraction  of  the  NVODS-accessible  data  sets 
via  LAS.  Installed  OPeNDAP  servers  for  PMEL  data  sets.  Very  high  participation. 

o.  Chris  Lynnes  -  Installed  servers  for  the  very  extensive  Goddard  DAAC  data  holdings. 
Moderate-to-high  participation. 

p.  Richard  Lawrence  -  Worked  with  Habermann  on  the  ESRI-OPeNDAP  data  connector.  Low 
participation. 

q.  Lola  Olsen  -  Developed  the  OPeNDAP  Data  Portal  to  the  Global  Change  Master  Directory. 
Worked  actively  with  the  project  to  describe  NVODS-accessible  data  sets  in  the  GCMD. 

High  participation. 

r.  Neville  Smith  -  Participated  very  actively  in  the  NVODS  National  Meetings.  Installed  a 
server  for  some  of  their  data  holdings.  Moderate -to-low  participation. 

s.  Ken  Tenore  -  Participated  in  one  of  the  annual  meetings.  Very  low  participation. 

3.  Partner  Long-Term  Commitment:  Co-Is  with  which  we  anticipate  a  close  continuing 
relationship  in  the  future  are:  W.  Nowlin,  Texas  A&M;  B.  Butman,  USGS;  R.  Raskin  for  Don 
Collins,  JPL  DAAC;  P.  Sharfstein  for  Jim  Cummings,  FNMOC;  B.  Domenico,  Unidata;  T. 
Habennann,  NGDC;  P.  Hamilton,  SAIC;  S.  Hankin,  PMEL;  L.  Olsen,  GCMD  and,  S.  Kempler 
for  Chris  Lynnes,  Goddard  DAAC.  These  colleagues  continue  to  serve  their  data,  add  new  data  to 
their  sites  or  develop  software  for  the  system  as  well  as  to  participate  in  on-going  planning  efforts 
and  in  the  submission  of  joint  proposals.  In  addition  to  these  project  Co-Is,  there  is  an  equal 
number  of  groups/individuals  who  were  not  Co-Is,  but  who  are  participating  at  a  similar  level. 

4.  Success  in  Project  Objectives:  All  of  the  objectives  identified  in  the  proposal  have  been  met. 
Specifically,  a  community-consensus  infrastructure  has  been  developed  for  the  consistent 
syntactic  delivery  of  data  via  the  network  directly  into  the  user’s  analysis  package;  the  system  has 
been  heavily  populated  with  oceanographic  and  meteorological  data  sets;  the  system  is  being 


heavily  used;  and,  a  vigorous  community  of  data  providers  and  software  developers  has  been 
established.  In  addition  to  the  tasks  initially  identified,  a  number  of  other  tasks  related  to  the 
system  have  been  undertaken  such  as  the  pilot  programs,  the  development  of  an  Aggregation 
Server  for  array  data,  the  development  of  an  Ancillary  Information  Services  capability  and 
interfaces  to  several  different  GIS  web-based  systems.  Our  success  in  meeting  these  objectives  is 
underscored  by  the  fact  that  several  large  projects  are  making  use  of  OPeNDAP  (a.k.a.  DODS)  as 
a  component  of  their  data  transport  layer: 

a.  NOMADS  http://nomads.ncdc.noaa.gov/, 

b.  DMAC/IOOS  http://dmac.ocean.us:8101/, 

c.  ESG  II  https://www.earthsvstemgrid.org/about/overview.isp; 

d.  COLA  http :// grads . i ges . or g/gr ads/gds/;  and, 

e.  GODAE  http://www.bom.gov.au/bmrc/ocean/GODAE/Meetings/7thIGST/data  sharing  paper.doc. 


